The Complexity of Generating Test Instances

Recently, Watanabe proposed a framework for testing the correctAbstract-1 ness and average-case performance of algorithms that purport to solve a given NP search problem efficiently on average with respect to some distribution on the instances. The idea is to randomly generate certified instances under some distribution that resembles the input distribution. Watanabe showed that unless RE = NE, test instances cannot be generated for some NP search problems. We further discuss Watanabe’s approach and show, as an upper bound, that test instances can be generated for every NP search problem with non-adaptive queries to an NP oracle. We also introduce Las Vegas and Monte Carlo types of test instance Abstract-2 generators and show that these generators can be used to find out (with high confidence) whether an algorithm is correct and efficient on average. It is shown that Monte Carlo generators can be easily constructed for all RP search problems and that Las Vegas generators exist for all ZPP search problems as well as for other problems such as prime factorization. On the other hand, we prove that Monte Carlo generators can only exist for problems in NP ∩ co-AM.